Papers with calibration method
Bag of Tricks for In-Distribution Calibration of Pretrained Transformers (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models (PLMs) often predict over-confidently. |
| Approach: | They propose to use ensemble learning and data augmentation to improve confidence calibration for PLMs by combining calibration techniques with a trade-off between accuracy and classification. |
| Outcome: | The proposed calibration method improves classification accuracy and confidence in pre-trained language models by combining several calibration techniques. |
Early Stopping Based on Unlabeled Samples in Text Classification (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to stop models from overfitting are based on a separate validation set, but in low resource settings, a small validation set may not be representative enough. |
| Approach: | They propose a method that uses unlabeled samples to estimate the class distribution of the unlabed samples. |
| Outcome: | The proposed method performs better than existing stop-methods in balanced and imbalanced data settings. |
Distinguishability Calibration to In-Context Learning (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have shown that pre-trained language models generate similar output embeddings which makes it difficult to discriminate for the prompt-based classifier. |
| Approach: | They propose a calibration method which rotates the embedding feature into a new metric space and adapts the ratio of each dimension to a uniform distribution. |
| Outcome: | The proposed method improves the distinguishability of learning embeddings on three datasets under various settings. |
Calibrating Structured Output Predictors for Natural Language Processing (2020.acl-main)
Copied to clipboard
| Challenge: | Several modern machine-learning based NLP systems can provide a confidence score with their output predictions. |
| Approach: | They propose a general calibration scheme for output entities of interest in NLP applications that can be used to calibrate confidence scores. |
| Outcome: | The proposed calibration scheme outperforms current calibration techniques for Named Entity Recognition, Part-of-speech tagging and Question Answering systems. |
Re-Examining Calibration: The Case of Question Answering (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing calibration methods do not provide significant gains in accuracy. |
| Approach: | They propose a new calibration metric that better captures whether the model assigns low confidence to wrong predictions and high confidence to correct predictions. |
| Outcome: | The proposed calibration method better captures whether the model assigns low confidence to wrong predictions and high confidence to correct predictions. |
Noisy-Labeled NER with Confidence Estimation (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies in deep learning have shown significant progress in named entity recognition (NER) . however, most existing works assume clean data annotation, while real-world data typically involve a large amount of noises. |
| Approach: | They propose a confidence estimation approach for named entity recognition using noisy labels using local and global independence assumptions. |
| Outcome: | The proposed method marginalizes out labels of low confidence with a CRF model and integrates it into a self-training framework for boosting performance. |
Compressing Language Models for Specialized Domains (2026.eacl-long)
Copied to clipboard
| Challenge: | Language models (LMs) excel at tasks across diverse domains, yet require substantial computational resources during inference. |
| Approach: | They propose a calibration method to improve the in-domain performance of compressed LMs in a post-training setting. |
| Outcome: | The proposed method outperforms existing methods on domain-specific tasks while preserving general performance. |
Complex Logical Query Answering by Calibrating Knowledge Graph Completion Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to solve complex logical queries are not well-calibrated . CKGC is lightweight and effective, allowing the model to quickly converge . |
| Approach: | They propose a method for calibrating KGC models to adapt to complex logical queries . they map the values of predictions of KGC to the range [0, 1] . |
| Outcome: | The proposed method can significantly boost model performance in complex logical query answering task. |
Fill In The Gaps: Model Calibration and Generalization with Synthetic Data (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing calibration methods negatively impact model accuracy due to the lack of diversity of validation data. |
| Approach: | They propose a calibration method that incorporates synthetic data without compromising accuracy. |
| Outcome: | The proposed method improves model accuracy on real data and reduces calibration error by 34% on four different tasks. |
Calibrated Seq2seq Models for Efficient and Generalizable Ultra-fine Entity Typing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | CASENT predicts ultra-fine entities mentioned in text into types with calibrated confidence scores. |
| Approach: | They propose a model that predicts ultra-fine entities with calibrated confidence scores for entity typing. |
| Outcome: | The proposed model outperforms existing models in terms of F1 score and calibration error while achieving 50 times faster inference speed. |
Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research demonstrates that multilingual encoder models are capable of zero-shot cross-lingual learning by using cloze-style prompts. |
| Approach: | They propose to reformulate input examples into cloze-style prompts to perform zero-shot multilingual tasks or linguistic probing by predicting label words at the masked token position. |
| Outcome: | The proposed method performs zero-shot multilingual tasks without updating parameters. |
Calibrating Model-Based Evaluation Metrics for Summarization (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in summary evaluation are based on model-based metrics to assess quality dimensions, such as completeness, conciseness, and faithfulness. |
| Approach: | They propose a general framework that generates individual and average proxy scores without relying on reference summaries, human annotations, or expensive model-based metrics. |
| Outcome: | The proposed framework outperforms baselines on seven datasets on continuous-value scenarios, such as summarization, but is applicable to discrete-value tasks, such QA. |